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(54) Situation awareness system 

(57) A situation awareness system Includes a plu- 
rality of cameras. Each camera acquires a sequence of 
images of a particular part of an area of interest. There 
is overlap between the parts so that the system can 
obtain depth information about objects in the area of 
interest. An analyzer identifies moving objects in the 



areas of interest, attributes of the moving objects, and 
events related to the moving objects. A display device 
displays the attributed objects and events as annotated 
graphic elements and alerts. 
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Description 

FIELD OF THE INVENTION 

[0001] The invention relates generally to a monitor- 
ing system rendering a synthetic display derived from 
multiple cameras mounted at various locations, and 
more particularly, to alerting a viewer on special situa- 
tions observed. 

BACKGROUND OF THE INVENTION 

[0002] Video monitoring systems are well known. In 
the case of vehicles, several types of monitoring sys- 
tems are in use. Some vehicles, e.g., busses, have 
cameras mounted so that the driver can view road 
areas beside or behind the bus. However, there is typi- 
cally only one camera, and the display merely shows 
exactly what the camera sees. There is no attempt to 
analyze the displayed image. These systems sirrply act 
as viewing mirrors for hard-to-see areas. Similarly, law 
enforcement vehicles may capture a historical record of 
the view from the front window. 

[0003] Some vehicles, such as computer controlled 
cars, also include sensors. The sensors detect poten- 
tially dangerous situations, such as, closing-up too rap- 
idly on another vehicle. A variety of sensors have been 
used, for example, sonar, lasers, and microwaves. 
These systems do not provide a general situation dis- 
play, rather they merely detect a few dangerous situa- 
tions. 

[0004] Radar and sonar systems can produce a sit- 
uation display, and sometimes do some amount of anal- 
ysis, for example, as in an air traffic control system. 
However, radar and sonar systems are not based on 
video images, but rather on the processing of reflected 
signals transmitted at specific frequencies. 
[0005] Several types of surveillance systems are 
known. Typically, the systems route multiple video 
streams to a central location. The video streams can be 
displayed on corresponding monitors. If the number of 
cameras is greater than the number of display stations, 
then the system usually displays camera views in 
sequence, or on operator demand. These type of sys- 
tems do not perform analysis, nor do these systems 
merge multiple streams Into a single situation display. At 
most they may tile multiple independent views on a sin- 
gle screen with time and location annotations. 
[0006] There are also systems that monitor specific 
places, such as escalators, elevators, toll gates, bank 
machines, and perimeter fences. In order to determine 
the occurrence of particular situations. Some of these 
systems may attempt to analyze the video in order to 
detect moving objects, for example, to extract a license 
number. However, these system typically do not com- 
bine information from multiple sources, nor do they gen- 
erate an overall situation display, nor synthesize an 
image from a different point of view. 



SUMMARY OF THE INVENTION 

[0007] The invention provides a situation aware- 
ness system which includes a plurality of cameras. 

5 Each camera acquires a sequence of images of over- 
lapping parts of an area of interest. An analyzer merges 
the sequences of images acquired by the plurality of 
cameras, and identifies moving objects in the area of 
interest. A display device displays the merged 

10 sequences of images, and information associated with 
the identified moving objects. 

[0008] In one aspect of the invention, the optical 
flow in temporally successive images of a single video 
stream are analyzed to generate motion fields. Spatially 

15 adjacent images of multiple video stream are registered 
to obtain depth images. The motion fields and depth 
images are segmented to generate partially attributed 
data objects. Using an application specific database 
and analysis, the partially attributed data objects are 

90 converted to fully attributed data objects and events 
which are displayed as annotated grapfiic elements and 
alerts. As one feature of the invention, the viewing orien- 
tation of the display is independent of the point of view 
of the cameras. 

25 

BRIEF DESCRIPTION OF THE DRAWINGSFigure 1 is 
a block diagram of an awareness system according to 
the invention; 

30 [0009] 

Rgure 2 is a block diagram of an analyzer synthe- 
sizer of the system of Figure 1 ; and 
Figure 3 is an example synthetic image generated 
35 by the system of Figure 1 . 

DETAILED DESCRIPTION OF PREFERRED EMBOD- 
IMENTSSystem Overview 

40 [0010] Figure 1 shows the situation awareness sys- 
tem 100 according to my invention. The system 100 
includes multiple video cameras 101-106. Each camera 
acquires a sequence of images as a video stream 115. 
Six cameras are shown, fewer or more can be used. 

45 Additional cameras can be provided for redundancy in 
the case of a camera failure, or obstruction. The cam- 
eras can be arranged to obtain a full 360 degree field of 
view of an area of interest around a vehicle. 
[001 1 ] For other applications, a smaller field of view 

so is suitable. The images provided by each camera over- 
lap parts of the area of interest such that stereoscopic 
techniques can be used to extract depth information. 
Wide angle lenses can be used to increase the amount 
of overlap without increasing the numljer of cameras. 

55 [0012] The output of the cameras, digitized video 
streams 115, is connected to an analyzer-synthesizer 
200. The analyzer-synthesizer 200, according to my 
invention, analyzes the video streams and generates a 
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synthetic display 300 on an output device 120. 
System Operation 

[001 3] tn an operational system, the cameras can 
be mounted on, for example, a vehicle 130 shown by 
dashed lines in Figure 1. The cameras can also be 
placed at other fixed or moving locations to observe the 
area of interest, generally 125, the areas in front o1 the 
various lenses. 

[0014] The analyzer-synthesizer 200 operates on 
the data of the multiple video streams in real-time. The 
analyzer portion extracts temporal and spatial data from 
the video streams to identify objects, and their 
attributes, such as size, position, and velocity. In addi- 
tion, relationships between the identified objects are 
determined, for example, two vehicles on intersecting 
courses. In other words the video streams are reduced 
to a relationship of attributed objects. The attributed 
objects are analyzed to detect events, for example, a 
possible collision, or a danger zone. The synthesizer 
portion generates the situation awareness display 300 
of the relationships of the attributed objects, and 
optional alerts related to the events. 
[0015] According to my invention, the situation 
awareness display 300 is entirely synthetic. In contrast 
with the prior art. I discard the video stream 115 after it 
Is analyzed. In addition, the synthesizer integrates infor- 
mation extracted from the multiple video streams into a 
single display 300. Furthermore, alert signals 140 may 
be generated when certain dangerous situations or 
events are recognized. The alert signals can be dis- 
played, or presented to some other output device 150. 
In an alternative embodiment, the alert signals 140 can 
initiate evasive collision avoidance action, for example, 
braking or slowing down. 

Synthesizer-Analyzer 

[0016] As shown in Figure 2, video streams 115 
from multiple cameras 101-106 are presented to the 
analyzer/synthesizer 200, via an A/D converter if neces- 
sary, as digital video data 201. Temporal and spatial 
information is extracted from the digital video data 201 . 
[0017] Optical flow analysis 210 is used to deter- 
mine motion fields 21 1 from images separated In time 
(?t), for example, from motion fields of successive 
frames in a single video sequence. 
[0018] Image registration 220 Is used to determine 
a depth Image 221 from Images overlapping in space 
(?x. ?y), for example, using frames taken of overlapping 
parts of the area of interest by multiple cameras. The 
depth image specifies the distance (?z) to each pixel In 
the image. 

[001 9] The motions fields and depth image are seg- 
mented to produce partially attributed data objects 231. 
For example, pixels having the same optical flow at the 
same depth are likely to be related to the same object. 



4 

Using both the optical flow and distances provides for a 
robust segmentation, particularly when the flow analy- 
sis Is done concurrently with the registration so the 
derived results (motion fields and depth values) corre- 

5 late with each other (215). 

[0020] The partial attributes can Include the size, 
position, velocity, direction of movement of the objects in 
thee-dimenslonal space. The objects are only partially 
attributed because other attributes that depend on addi- 

10 tional knowledge, such as the exact Identity of the 
objects have not yet been determined. 
[0021] The partially attributed data objects 231. in 
conjunction with an application specific database 239 
can be analyzed 240 to generate fully attributed data 

75 objects 241 and events 242. For example, a one-sided 
view of a semi-trailer is sufficient to deduce the entire 
shape of the object. Various kinds of template matching 
schemes can be used to fully identify specific commonly 
occurring objects, such as. other vehicles, pedestrians. 

20 bicycles, trucks, and the like. In a vehicle application, 
the features may also Include lane dividers, side walks, 
stop signs, guard rails, curbs, buildings, fences, and so 
forth. 

[0022] The events 242 can be generated by analyz- 
es ing the relationships among the attributed objects, for 
example, a potential collision situation, a car drifting off 
the road, or a fading light situation. Additional sensors 
249 can also be used to enrich the number of events 
that can be detected. 
30 [0023] A synthesizer 250 converts the fully attrib- 
uted data objects 241 to annotated graphic elements 
251 and alerts 252. The last step renders 260 the 
graphic elements 251 and alerts 252, 

35 Display 

[0024] Many different types of situation displays are 
possible. The display 300 in Figure 3 shows a bird's eye 
view of the area of Interest with the vehicle 310 on which 

40 the situation awareness device is mounted, located at a 
fixed orientation near the center of the display, and 
annotated objects moving relative to the point of view. 
Note, the view is totally synthetic and orthogonal to the 
view seen by the cameras. 

45 [0025] Certain other image features are shown as 
well, such as pedestrian lane crossing 320, buildings 
330, other traffic 340. a bicycle 350. and so forth. 
[0026] Arrows 301 can be used to show the direc- 
tion of movement of objects that are not- stationary. 

so Determining the orientation of the arrows requires an 
active analysis, as opposed to passively displaying the 
output of the cameras as done in the prior art. 
In an area of Interest where sufficient ambient light can 
not be assured, my invention can be extended by Includ- 

55 Ing active illumination. In some situations K could benefit 
from using Infrared light, either to see in the dark without 
requiring active illumination or as inoffensive active illu- 
mination. In situations such as fog. where visibility Is 
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poor, my invention can operate by carefully selected 
wavelengths or strobed light sources appropriately syn- 
chronized with the shutter of the cameras so as to focus 
on objects of interest and reject other scattered light. 
[0027] In one embodiment of my invention, the ana- 
lyzing step 240 can receive secondary data 238. In a 
vehicle application, the data can include vehicle velocity, 
or position as obtained from a GPS receiver. With the 
vehicle's velocity, the analysis can be improved and sim- 
plified. Positional data enables the use of maps on the 
display, and actual street and place names. 
[0028] In another embodiment, the display 300 is 
under user control. For instance, in a building surveil- 
lance application, the user supply control signals 239 to 
alter the way that the annotated graphic elements and 
alerts are displayed, or to change the orientation of the 
point of view. It is also possible to transmit the alerts and 
graphic elements to a remote location. For instance, 
while walking toward a parked vehicle, the operator can 
view, on a portable display device, the area of interest in 
the vicinity of the vehicle from a safe, location. 
[0029] In addition, multiple vehicles can exchange 
situation information with each other to enhance the 
scope of the display. Other areas where the invention 
can be used include airports, waterways, and the like. 
[0030] This invention is described using specific 
terms and examples. It is to be understood that various 
other adaptations and modifications may be made 
within the spirit and scope of the invention. Therefore, it 
is the object of the appended claims to cover all such 
variations and modifications as come within the true 
spirit and scope of the invention. 

Claims 

1. A real-time situation awareness system, compris- 
ing: 

a plurality of cameras acquiring a plurality of 
video streams of overlapping parts of an area 
of interest; 

analyzer means for reducing the plurality of 
video streams to attributed data objects and 
events; and 

synthesizer means for rendering the attributed 
data objects and events as annotated graphic 
elements and alerts on an output device. 

2. The system of claim 1 further comprising: 

means for temporally analyzing an optical flow 
in successive images of a single video stream 
to generate motion fields: 
means for spatially registering adjacent images 
of multiple video stream to obtain depth 
images; and 

means for segmenting the motion fields and 
depth images to generate partially attributed 



data objects. 

3. The system of claim 2 further comprising: 

5 means for analyzing the partially attributed 

data objects using an application specrfic data- 
base to generate fully attributed data objects 
and events. 

10 4. The system of claim 3 further comprising: 

sensors providing the analyzing step with sec- 
ondary data and signals. 

15 5. The system of claim 1 wherein the synthesizer 
means produces a display having a point of view 
substantially orthogonal to the point of view of the 
cameras. 

so 6. The system of claim 1 wherein the area of interest 
is a panoramic scene. 

7. The system of claim 1 wherein annotations for the 
graphic elements include directions of movement. 

8. The system of claim 5 wherein user control signals 
determine the display. 

9. A method for generating a real-time situation 
30 awareness display, comprising the steps of: 

acquiring a plurality of video streams of over- 
lapping parts of an area of interest; 
reducing the plurality of video streams to attrib- 
35 uted data objects and events; arxi 

rendering the attributed data objects and 
events as annotated graphic elements and 
alerts on an output device. 

40 10. The method of claim 9 further comprising: 

temporally analyzing an optical flow in succes- 
sive images of a single video stream to gener- 
ate motion fields; 
45 spatially registering adjacent images of multi- 

ple video stream to obtain depth images; and 
segmenting the motion fields and depth images 
to generate partially attributed data objects. 

so 11. The method of claim 10 further comprising: 

analyzing the partially attributed data objects 
using an application specific database to gen- 
erate fully attributed data objects and event. 
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